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Introduction 

At  the  Forum’s  founding  meeting,  Joseph 
Weizenbaum  sketched  out  the  limits  of  respon¬ 
sible  computer  use  in  the  following  terms:  one 
should  not  do  things  with  computers  that  one 
could  not  accept  responsibility  for  doing  without 
computers.  I  should  like  to  accept  that  as  our 
motto,  and  yet  at  the  same  time  I  feel  it  to  be 
insufficient.  As  I  see  it,  we  here  in  the  Forum 
must  construct  a  detailed  and  technically  sound 
set  of  arguments  which  will  enable  us  to  assess 
individual  concrete  developments  according  to 
our  own  values  and  professional  judgments. 

I  don’t  mean  by  this  a  set  of  rules  permitting  a  sort 
of  grading  system  (responsible/irresponsible): 
the  assessment  will  always  be  a  dynamic  social 
process  allowing,  in  the  case  of  conflicts,  scope 
for  other  viable  alternatives.  To  ensure  that  this 
scope  is  utilized  in  accordance  with  our  own  par¬ 
ticular  views  and  values,  it  is  up  to  us  to  provide 
a  contribution  to  the  discussion  setting  out  these 
views. 
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In  what  follows  I  shall  attempt  to  outline  a 
number  of  points  relating  to  the  limits  of  re¬ 
sponsible  computer  use.  I  shall  distinguish 
here  between  several  different  categories:  I 
see  the  technical  limits  of  responsible  com¬ 
puter  use  — i.e.,  from  the  point  of  view  of  the 
profession  itself— where  computers  are  util¬ 
ized  as  a  result  of  misguided  trust  in  the 
capabilities  of  computer  programs.  My  aim 
here  is  to  call  into  question  the  exaggerated 
claims  of  our  particular  field.  I  don’t  wish  to 
merely  generalize  about  computer  errors 
which  might  be  avoided  with  the  help  of  an  im¬ 
proved  technology  or  methodology,  but  rather 
to  draw  attention  to  the  fundamental  limits  of 
this  field  as  suggested  by  our  professional  ex¬ 
perience  as  computer  scientists. 

I  see  the  human  limits  of  responsible  computer 
use  — i.e.,  from  the  point  of  view  of  human 
interaction — where  computers  are  used  as  a 
result  of  the  misguided  equating  of  people  with 
machines.  The  areas  in  question  here  are 
those  in  which  human  interaction  is  hindered 
as  a  result  of  people’s  being  replaced  by 
computer  programs;  in  which  human  experi¬ 
ence  dwindles,  human  care  ceases  to  exist 
and  social  networks  are  destroyed.  We  as 
computer  scientists  must  reject  any  attempt  to 
raise  the  computer  to  the  level  of  man’s  “part- 
ner. 

Finally,  the  ethical  and  political  limits  of  com¬ 
puter  application  must  be  drawn  where  (to 
paraphrase  Weizenbaum)  the  attempt  is 
made  to  do  things  with  computers  that  ought 
not  be  done  without  them.  We  cannot  shirk  re¬ 
sponsibility  for  things  that  happen  as  a  result  of 
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our  computer  programs.  The  use  of  computers 
has  made  it  possible  to  split  our  sense  of  respon¬ 
sibility  to  a  greater  extent  than  has  previously  been 
the  case  with  other  complex  technological  applica¬ 
tions.  Between  the  preplanning  (programming)  of 
events  and  courses  of  action  and  their  occurrence, 
the  computer  has  now  been  interpolated  as  a  non¬ 
human  element.  Nevertheless,  the  responsibility 
for  what  computers  “do”  still  rests  with  the  people 
who  produce  and  program  them,  who  plan  and 
govern  their  application ;  which  means  that  we  bear 
this  responsibility  too. 

In  the  following  pages,  I  shall  be  using  general 
principles  to  examine  these  limits  in  greater  detail. 
In  our  work  in  the  Forum  they  can  be  used  primarily 
to  throw  critical  light  on  questionable  develop¬ 
ments  in  the  military  sector.  But  our  responsibility 
does  not  end  there:  we  must  help  to  lay  down  the 
limits  of  responsible  computer  use  in  all  areas  of 
society.  My  main  concern,  though,  is  that  we  use 
this  discussion  in  the  Forum  to  encourage  a  proc¬ 
ess  of  rethinking  among  computer  scientists,  each 
of  us  in  our  own  particular  sphere  of  work  and  in 
accordance  with  our  own  responsibilities,  so  as  to 
enable  us,  through  our  combines  efforts,  to  lay  the 
professional  foundations  for  a  computer  science 
more  closely  geared  to  human  values. 

Technical  Limits  of 
Responsible  Computer  Use 

Although,  in  my  view,  this  question  requires  highly 
detailed  argumentation,  the  present  paper  can 
represent  no  more  than  a  tentative  attempt  at 
coming  to  terms  with  the  issue.  I  hold  that  the  limits 
described  below  are  inherent  in  the  nature  of  our 
work,  in  other  words  they  may  be  shifted  by  tech¬ 
nological  and  methodological  improvements,  but 
never  eliminated  altogether. 

All  models  of  reality  anchored  in  a  program  are,  by 
their  very  nature,  reductionist.  This  means  that  in 
specifying  or  implementing  programs  we  are  al¬ 
ways  forced  to  model  reality  in  all  its  diversity  using 
a  finite  number  of  selected  objects  (or  classes  of 
objects),  their  selected  but  henceforth  fixed  char¬ 
acteristics,  as  well  as  the  operations  permitted  on 
these  objects.  In  so  doing,  we  always  create  an 
artificial,  closed  microcosm  which  alone  is  decisive 
for  the  functioning  of  the  program.  In  reality,  we 


must  always  be  prepared  for  unforeseen  events  or 
objects  with  new  characteristics;  the  limit  imposed 
by  this  process  of  modelling  with  the  help  of  pro¬ 
grams  is  intrinsically  insurmountable. 

The  intelligence  anchored  in  a  computer  differs 
fundamentally  from  human  intelligence  in  the  fol¬ 
lowing  respects:  it  always  relies  on  a  model  which 
is  based  on  fixed  rules  (  in  some  cases  with 
additional  rules  for  expanding  these  dynamically). 
Human  intelligence,  on  the  other  hand,  incorpo¬ 
rates  as  essential  elements  an  appreciation  of  the 
situation  in  hand,  together  with  past  experiences 
relating  to  it,  and  experienced  needs  and  values. 
Artificial  intelligence  is  not  rooted  in  the  senses  and 
the  body,  and  therefore  lacks  the  association  with 
feeling  and  acting — for  us  ever  present  and  taken 
for  granted — an  association  without  which  reality 
is  devoid  of  human  intelligence. 

Decisions  on  the  basis  of  program  results  should 
therefore  not  be  taken  by  other  programs  unless  it 
is  ensured  that  human  beings  with  sufficient  com¬ 
petence  for  taking  these  decisions  are  able  to 
intervene  as  the  responsible  parties  in  the  deci¬ 
sion-making  process.  Only  human  beings  can 
assess  results  and  fit  them  meaningfully  into  the 
ever-changing  interpretational  context  which  itself 
emerges  as  a  result  of  human  communication. 

We  shall  never  succeed  in  totally  eliminating  pro¬ 
gram  errors  because  to  err  is  human.  But  erring  is 
not  only  a  human  weakness,  it  is  also  a  human 
strength,  being  as  it  is  so  closely  bound  up  with  the 
capacity  for  learning  and  finding  unconventional 
solutions.  Any  given  program  can  be  no  more  than 
the  result  of  limited  human  insight  at  a  given  point 
in  time.  We  all  know  that  errors  in  programs  may 
have  a  wide  variety  of  causes:  insufficient  under¬ 
standing  of  the  problem  at  hand,  inadequate  com¬ 
munication,  equivocal  agreements,  ignorance  of 
the  actual  application  context  of  programs,  insuffi¬ 
cient  knowledge  about  the  available  resources, 
errors  in  reasoning,  carelessness,  lack  of  or  differ¬ 
ence  in  motivation.  We  shall  not  be  able  to  elimi¬ 
nate  a  single  one  of  these  by  improvements  in 
technology  or  methodology.  Program  errors,  then, 
laydown  one  intrinsic  limit  of  computer  application: 
in  other  words,  computers  can  only  be  used  re¬ 
sponsibly  where  the  possibility  of  locating  and 
eliminating  program  errors  exist. 
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I  see  one  intrinsic  limit  of  responsible  computer 
application  in  the  lack  of  transparency  of  large 
program  systems — not  that  I  claim  to  be  able  to 
specify  the  maximum  acceptable  size  of  such 
systems — and  the  consequent  impossibility  of 
tracing  the  causes  of  located  errors  and  reliably 
eliminating  them  in  time.  I  do  not  consider  this  limit 
to  be  surmountable  by  improving  specification, 
programming,  or  documentation  methods;  in¬ 
stead,  it  requires  a  conscious  confinement  to 
small,  loosely  inter-connected  program  systems 
separated  by  human  beings  acting  in  an  interpre¬ 
tative  and  evaluative  capacity  who  are  familiar  with 
the  functioning  of  the  sufficiently  small  programs 
and  in  a  position  to  alter  these,  should  an  error 
occur,  or  compensate  for  it  by  manual  circumven¬ 
tion  of  the  program. 

It  is  imperative  that  programs  are  made  to  fit  the 
actual  demands  of  the  application  context  by  trial 
operation  under  real  conditions  and  subsequent 
adaptation.  I  do  not  simply  mean  by  this  the  elimi¬ 
nation  of  program  errors  in  accordance  with  a 
given  specification,  but  rather  the  interplay  be¬ 
tween  the  program  and  the  technical  or  social 
system  surrounding  it.  Particularly  in  the  case  of 
the  increasingly  common  embedded  systems,  the 
desired  functionality  of  programs  cannot  be  estab¬ 
lished  unequivocally  and  context-free:  instead,  the 
proper  functioning  of  the  program  can  only  be 
understood  in  the  context  of  the  surrounding  sys¬ 
tem.  Where  a  trial  in  this  context  under  real 
conditions  is  not  feasible,  it  is  not  possible  to 
assess  the  adequacy  of  the  programs  either;  it  is 
therefore  irresponsible  to  make  human  beings 
dependent  on  the  functioning  of  such  programs 
when  important  issues  are  at  stake. 

Finally,  I  see  one  intrinsic  limit  of  responsible  com¬ 
puter  application  in  the  susceptibility  to  error  and 
failure  inherent  in  computer  operation.  No  matter 
what  organizational  measures  we  may  take,  we 
shall  not  be  able  to  eliminate  the  possibility  of 
machines  operating  defectively  at  some  time  or 
other;  of  programs  being  erroneously  fed  the 
wrong  data;  or  even  of  correctly  functioning,  and 
for  their  application  context  quite  adequate  pro¬ 
grams  producing  incorrect  results  owing  to  other 
unforseeable  events,  in  the  last  analysis,  only 
human  beings  are  capable,  through  assessment 
of  the  overall  situation  on  the  basis  of  numerous 
individual  and  differing  perceptions,  of  distinguish¬ 


ing  between  real  events  and  those  merely  simu¬ 
lated  with  the  help  of  data  and  without  any  real 
background. 

Human  Limits  of 
Responsible  Computer  Use 

What  I  have  to  say  here  appears  to  me  to  be  self- 
evident;  however,  in  view  of  current  developments 
in  computer  application  in  our  society,  I  feel  it  nec¬ 
essary  to  state  the  obvious. 

When  using  computers,  the  actions  of  computa¬ 
tion,  data  processing  and  symbol  manipulation  are 
severed  from  all  other  ways  of  dealing  with  reality 
available  to  man,  and  are  alone  effective.  In  my 
view  this  is  only  defensible  if: 

•  computers  are  used  in  applications  where  , 
computation  and  symbol  manipulation  are 
adequate  ways  of  dealing  with  reality; 

•  there  remains  sufficient  scope  outside  the 
computer  application  for  other  human 
faculties  and  forms  of  experience  not  to 
degenerate. 

In  current  practice  these  two  points  are,  in  my  view, 
given  no  or  far  too  little  consideration.  And,  in  this 
respect,  computer  science  relies  on  kindred 
mechanistic  theories  in  the  humanities  which,  fol¬ 
lowing  the  procedures  adopted  in  the  natural  sci¬ 
ences,  attempt  to  analyse  our  inner  world  in  terms 
of  clearly  demarcated  sub-areas.  This  is  done  by 
confinement,  for  example,  to  statements  about  the 
cognitive  faculties  of  man  in  order  to  obtain  con¬ 
crete  results.  It  is  not  my  intention  here  to  question 
whether  a  separative  consideration  of  this  sort  is 
scientifically  meaningful;  but  I  feel  it  essential  to 
stress  that  it  does  not  follow  from  this  separative 
approach  in  scientific  consideration  that  these 
areas  may  ever  be  separated  in  reality  too. 

We  as  human  beings  alternate  between  mental 
states  in  which  we  compute,  feel,  process  informa¬ 
tion,  remember  or  dream,  skipping  back  and  forth 
at  any  time  with  such  facility  that  we  are  scarcely 
aware  of  it.  Computer  science,  on  the  other  hand, 
nurtures,  in  my  view,  an  image  of  man  in  which  the 
diversity  of  human  experience  and  action  is  re¬ 
duced  to  information  processing  in  the  technical 
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sense  and  to  symbolic  object  manipulation.  Here, 
I  regard  the  following  points  as  particularly  prob¬ 
lematic; 

•  the  postulated  structural  similarity  of  man 
and  computer  as  information-processing 
systems: 

•  the  consequent  view  that  human  prob¬ 
lem-solving  behavior  is  governed  by  rules 
and  is  therefore  classifiable  and  predict¬ 
able; 

•  the  already  mentioned  assumption  that  it 
is  possible  to  disregard  an  essential  and 
ever-present  feature  of  man’s  mental 
make-up,  i.e.,  that  his  faculties  of  compu¬ 
tation  and  symbol  manipulation  are  inextri¬ 
cably  linked  with  other  forms  of  experi¬ 
ence,  such  as  feeling  or  acting,  which  are 
governed  both  by  the  connection  between 
the  mind  and  body  and  by  the  circum¬ 
stances  relating  to  the  particular  unique 
situation  in  hand  and  to  its  interpretation  by 
the  experiencing  human  being. 

At  this  point  I  do  not  wish  to  go  further  into  rival 
theories  in  the  humanities  in  which  other  views  of 
man  are  opposed  to  the  mechanistic  one;  nor  do  I 
wish  to  examine  the  empirical  evidence  for  or 
against  the  mechanistic  view  of  man.  But  I  do 
emphatically  desire  to  disassociate  myself  from 
this  view  as  the  sole  basis  for  the  design  and 
application  of  computer-aided  systems  since  it 
conflicts  with  my  experience  of  myself  and  in 
dealings  with  other  human  beings  in  a  number  of 
essential  points;  and,  what  is  more,  it  threatens 
certain  qualities  in  interpersonal  relations  which 
are  for  me  indispensable. 

I  also  feel  that  we  must  draw  the  limits  of  respon¬ 
sible  computer  use  by  consciously  adapting  a 
richer  view  of  man  and  using  this  as  the  basis  for 
our  work.  This  involves  both  examining  theories 
about  man  which  conflict  with  the  mechanistic 
view,  and  also — and  this  is  of  particular  impor¬ 
tance — recognizing  the  difference  between  man 
and  machine  in  our  own  experience  and  in  our 
relation  with  other  human  beings,  so  as  to  embody 
it. 

The  points  mentioned  above  appear  to  me  to  be  of 


significance  for  a  number  of  specific  concrete 
developments  in  computer  science  and  computer 
application  of  which  the  following  may  stand  as 
examples: 

The  area  of  human-machine  interaction.  Here,  we 
must  make  it  clear  that  between  human  beings  and 
machines  there  can  be  no  communication  in  the 
proper  sense  of  the  word,  and  no  dialogue  can  be 
carried  on;  that  the  computer  will  never  be  man’s 
partner:  and  that  we  must  therefore  ensure  that 
communication,  dialogue  and  partner-like  coop¬ 
eration  between  human  beings  remains  possible 
in  jobs  involving  work  at  computer  terminals. 

Changes  in  work  processes  through  computer 
application.  We  must  see  to  it  that  the  role  of 
human  beings  in  computer  aided  systems  does  not 
degenerate  into  meaningless  routine  work  and 
computer  operation,  but  rather  that  meaningful 
human  activities  are  adequately  supported  by  the 
computer. 

The  use  of  computers  in  areas  where  interpersonal 
relations  are  of  paramount  importance.  Here,  we 
must  insist,  for  example,  that  the  human  relation¬ 
ship  between  teacher  and  student  cannot  be  re¬ 
placed  by  computer-aided  instruction;  that  human 
care  for  the  elderly  cannot  be  substituted  by  the 
use  of  geriatric  robots,  as  proposed  by  certain 
computer  scientists:  that  jurisdiction  is  more  than 
information  processing  with  reference  to  the  par¬ 
ticular  case  in  hand  and  the  relevant  laws;  that  the 
assessment  of  colleagues’  achievements  cannot 
be  reduced  to  the  numerical  evaluation  of  selected 
performance  characteristics. 

The  invasion  of  leisure  time  and  family  life  by  the 
computer.  Here  we  are  quite  justified  in  posing  the 
question  as  to  what  extent  the  continued  and  one¬ 
sided  preoccupation  with  computers  allows  other 
human  capabilities,  which  are  thereby  neglected, 
the  chance  of  developing  harmoniously. 

The  drawing  of  a  line  between  man  and  machine  is 
not,  in  the  last  analysis,  imposed  from  without,  but 
is  rather  inherent  in  man  as  a  thinking  and  acting 
being  in  who,  machine-like  and  non-machine-like 
elements  interact.  Through  our  constant  confron¬ 
tation  with  the  computer  we  risk  being  encouraged 
to  think  and  act  more  like  machines,  our  other 
faculties  going  to  waste.  If  we  wish  to  avert  the 
negative  effects  of  the  computer  on  our  environ- 
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ment,  we  must  first  of  all  come  to  terms  with  the  ma¬ 
chine  in  our  own  selves. 

Ethical  and  Political  Limits  of 
Responsible  Computer  Use 

Wherever  computers  are  used  there  is  a  danger  of 
their  being  utilized  to  promote  the  interests  of 
individuals  or  specific  groups  in  order  to  do  things 
with  computers  that  ought  not  to  be  done  without 
them.  I  shall  therefore  not  attempt  to  survey  the 
areas  which  are  particularly  at  risk,  but  rather 
outline  the  ways  in  which  computers  may  be  used 
to  circumvent  accepted  ethical  or  political  limits. 

•  With  the  help  of  computers  the  attempt 
can  be  made  to  conceal  measures  whose 
open  application  would  be  impossible  on 
the  basis  of  currently  accepted  values  and 
legislation.  Instances  of  such  utilization 
are  the  surveillance  and  control  of  the 
individual  by  the  state  and  private  organi¬ 
zations  and  enterprises. 

•  With  the  help  of  computers  one  can  set 
out  to  circumvent  ethical  action — which  is 
exclusive  to  man — so  as  to  force  pre¬ 
planned  events  to  occur  contrary  to  man’s 
sense  of  responsibility.  This  is  evidenced, 
for  example,  by  considerations  as  to  the 
necessity  of  triggering  off  the  destruction 
of  a  city  (in  the  case  of  a  nuclear  war)  by 
means  of  a  computer  on  the  grounds  that 
human  beings  would  probably  be  unwill¬ 
ing,  even  under  stress  and  acting  under 
orders,  to  carry  out  a  command  of  this  sort. 

•  With  the  help  of  computers  one  can  at¬ 
tempt  to  shirk  responsibility  for  the  occur¬ 
rence  of  programmed  events  by  delegat¬ 
ing  the  responsibility  to  the  machine.  Al¬ 
though  the  computer  stands  as  a  nonhu¬ 
man  element  between  programming  and 
execution,  the  responsibility  for  the  effects 
of  applying  our  programs  rests  with  us  as 
human  beings. 

•  Finally,  one  can  with  the  help  of  comput¬ 
ers  set  out  to  exceed  the  limits  of  human 
responsibility  altogether.  More  specifi¬ 
cally,  we  are  today  in  the  process  of  plan¬ 


ning  the  application  of  programs  which 
would  be  capable,  on  occurrence  of  a  local 
problem,  of  wiping  out,  without  human 
intervention,  all  life  on  this  planet. 

Gunther  Anders  defines  the  permissible  limits  for 
human  products  as  follows:  “Man,  being  as  he  is 
at  liberty  to  produce  things,  probably  has  no  crite¬ 
rion  of  this  sort  to  go  by:  that  is,  unless  he  takes 
himself  as  criterion;  in  other  words,  defines  the  limit 
as  being  reached  at  the  point  where,  being,  so  to 
speak  ‘smaller  than  himself,’  he  ‘cannot  keep  up 
with  himself,’  /.e.,  can  no  longer  cope  with  his  own 
products.  As  is  the  case  today.”  (Excerpt  from  “Die 
Antiquiertheit  des  Menschen,"  Vol.  1 ,  Uber  die 
Seele  im  Zeitalter  der  zweiten  industriellen  Revo¬ 
lution,  C.H.  Beck,  Munich  1980,  p.  45.)  □ 
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